Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(al2): use ecr-credential-provider for public.ecr.aws in 1.27+ #1949

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

feat(al2): use ecr-credential-provider for public.ecr.aws in 1.27+ #1949

wants to merge 1 commit into from

Conversation

cartermckinnon
Copy link
Member

Issue #, if available:

Discussed in #1317

Description of changes:

This allows authenticated public.ecr.aws pulls, to avoid the bandwidth limits for anonymous requests.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

@cartermckinnon
Copy link
Member Author

/ci

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 4, 2024

@cartermckinnon roger that! I've dispatched a workflow. 👍

@cartermckinnon cartermckinnon changed the title Public ecr al2 templates(al2): use ecr-credential-provider for public.ecr.aws in 1.27+ Sep 4, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Sep 4, 2024

@cartermckinnon the workflow that you requested has completed. 🎉

AMI variantBuildTest
1.23 / al2success ✅success ✅
1.24 / al2success ✅success ✅
1.25 / al2success ✅success ✅
1.26 / al2success ✅success ✅
1.27 / al2success ✅success ✅
1.28 / al2success ✅success ✅
1.29 / al2success ✅success ✅
1.30 / al2success ✅success ✅

@cartermckinnon
Copy link
Member Author

Folks may not have ECR Public permissions in their node's IAM role, so I need to make sure that kubelet will still attempt a pull if the cred provider returns an error -- I think that's the case IIRC

@sidewinder12s
Copy link

Any update on this? @cartermckinnon

@cartermckinnon cartermckinnon changed the title templates(al2): use ecr-credential-provider for public.ecr.aws in 1.27+ feat(al2): use ecr-credential-provider for public.ecr.aws in 1.27+ Nov 30, 2024
@mselim00
Copy link

Original assumption seems to be correct, the node still manages to pull the image after the initial error for the ecr-public pull

Feb 10 00:09:50 <NODE_NAME> kubelet[3372]: E0210 00:09:50.270753    3372 plugin.go:235] Failed getting credential from external registry credential provider: error execing credential provider plugin ecr-credential-provider for image public.ecr.aws/nginx/nginx: exit status 1: I0210 00:09:49.977805 10767 main.go:100] Getting creds for public registry
Feb 10 00:09:50 <NODE_NAME>  kubelet[3372]: E0210 00:09:50.269647   10767 main.go:262] Error running credential provider plugin: operation error ECR PUBLIC: GetAuthorizationToken, https response error StatusCode: 400, RequestID: 9cdb9c8d-4615-48d6-a24a-d5dbd6a77066, api error AccessDeniedException:User: arn:aws:sts::<ACCOUNT_ID>:assumed-role/<MINIMAL_NODE_ROLE>/<INSTANCE_ID> is not authorized to perform: ecr-public:GetAuthorizationToken on resource:* because no identity-based policy allows the ecr-public:GetAuthorizationToken action
Feb 10 00:09:50 <NODE_NAME>  kubelet[3372]: I0210 00:09:50.691738    3372 kubelet.go:2483] "SyncLoop (PLEG): event for pod" pod="default/nginx" event={"ID":"c6436473-30a3-41a4-97ca-6a65add5f409","Type":"ContainerStarted","Data":"0e9d0f92e1c228406dfb12c8b1545c6118a79c6d7982ee2bff99b0ac81ea735c"}
Feb 10 00:09:54 <NODE_NAME> kubelet[3372]: I0210 00:09:54.703589    3372 kubelet.go:2483] "SyncLoop (PLEG): event for pod" pod="default/nginx" event={"ID":"c6436473-30a3-41a4-97ca-6a65add5f409","Type":"ContainerStarted","Data":"eeb7b1f165cfcdd88f4c0856cf69e91fbd852f6bb576e68075de4b1ba2cce625"}

For maintenance purposes, we should consider moving https://github.com/awslabs/amazon-eks-ami/blob/main/templates/al2/runtime/bootstrap.sh#L188 here as well, with the new TODO being to move this logic into the config file after 1.26 reaches end of support.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@mselim00 mselim00

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@cartermckinnon @sidewinder12s @mselim00